{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "363a6974-810c-41c5-84da-4751a92fb72b",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Pruning and Distillation of Llama 3.1 model with NeMo Framework"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6d4ed6d-8ecd-4647-bd0a-e48fec64c199",
   "metadata": {},
   "source": [
    "This notebook showcases performing pruning and distillation on **Llama 3.1-8B-Instruct** with the [WikiText-103-v1](https://huggingface.co/datasets/Salesforce/wikitext/viewer/wikitext-103-v1) dataset using NeMo Framework. The [WikiText-103-v1](https://huggingface.co/datasets/Salesforce/wikitext/viewer/wikitext-103-v1) language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia. \n",
    "\n",
    "For this demonstration, we will perform a light finetuning procedure on the `Meta Llama 3.1 8B Instruct` teacher model to generate a finetuned teacher model. This finetuned teacher model will then be trimmed to create a depth-pruned model `4b_trimmed_model.nemo` that will serve as a starting point for distillation to a final 4B model. \n",
    "\n",
    "> We are using models utilizing the `meta-llama/Meta-Llama-3.1-8B` tokenizer for this demonstration.\n",
    "\n",
    "> `NOTE:` Ensure that you run this notebook inside the [NeMo Framework container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo) which has all the required dependencies. \n",
    "\n",
    "**Instructions are available in the associated tutorial README to download the model and the container.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d0dc714-5bbf-4266-805a-9841ff486c05",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install --upgrade ipywidgets notebook\n",
    "!pip install datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2658505d-7990-40a5-a269-866ddd8a0181",
   "metadata": {
    "tags": []
   },
   "source": [
    "---\n",
    "## Prerequisites\n",
    "Ensure you have the following -\n",
    "1. **Get the teacher model**: Download the `Meta Llama 3.1 8B Instruct .nemo` model. You must follow the instructions in the associated README to download and mount the folder to the NeMo FW container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a30cfe8a-87a8-4511-be5f-e20d7fe558d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls /workspace/llama-3_1-8b-instruct-nemo_v1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "251a670e-9636-4807-bc98-a91c6137454d",
   "metadata": {},
   "source": [
    "2. **Set the Hugging Face Access Token**: You can obtain this from your [Hugging Face account](https://huggingface.co/docs/hub/en/security-tokens). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47d7887d-b582-4a1e-81cd-fdc1be8d9afb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from huggingface_hub import login\n",
    "login(token=\"<YOUR_HF_ACCESS_TOKEN>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5384e9a-6c40-4454-abe8-413ad9d5db96",
   "metadata": {},
   "source": [
    "3. **Obtain the dataset**: Generate the `wikitext-{train/val/test}.jsonl` splits after loading the [WikiText-103-v1](https://huggingface.co/datasets/Salesforce/wikitext/viewer/wikitext-103-v1) dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b420bd44-3628-45e2-92e7-df38f72a658a",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Split into train, test and val files\n",
    "\n",
    "import json\n",
    "import os\n",
    "from datasets import load_dataset\n",
    "\n",
    "# Load the WikiText-103 dataset\n",
    "dataset = load_dataset(\"wikitext\", \"wikitext-103-v1\")\n",
    "\n",
    "# Define the destination folder\n",
    "data_folder = 'wikitext-data'\n",
    "os.makedirs(data_folder, exist_ok=True)\n",
    "\n",
    "# Define file paths and destination paths\n",
    "file_paths = {\n",
    "    'train': os.path.join(data_folder, 'wikitext-train.jsonl'),\n",
    "    'validation': os.path.join(data_folder, 'wikitext-val.jsonl'),\n",
    "    'test': os.path.join(data_folder, 'wikitext-test.jsonl')\n",
    "}\n",
    "\n",
    "# Function to save dataset split to a JSONL file\n",
    "def save_to_jsonl(file_path, data):\n",
    "    with open(file_path, 'w') as file:\n",
    "        for item in data:\n",
    "            file.write(json.dumps(item) + '\\n')\n",
    "\n",
    "# Define splits\n",
    "splits = [\"train\", \"validation\", \"test\"]\n",
    "\n",
    "# Save splits to JSONL files and calculate their sizes\n",
    "for split in splits:\n",
    "    if split in dataset:\n",
    "        save_to_jsonl(file_paths[split], dataset[split])\n",
    "    else:\n",
    "        print(f\"Split {split} not found in the dataset.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0185a0a9-904d-46de-a450-db4c84c4cde4",
   "metadata": {
    "tags": []
   },
   "source": [
    "---\n",
    "##  Step-by-step instructions\n",
    "\n",
    "This notebook is structured into five steps:\n",
    "1. Prepare the dataset\n",
    "2. Finetune the teacher on the dataset\n",
    "3. Prune the finetuned-teacher model to create a student\n",
    "3. Distill knowledge from teacher into student\n",
    "4. Display the validation loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf1d41ff-2cba-4efc-84e3-7d713df0cdb8",
   "metadata": {},
   "source": [
    "### Step 1: Prepare the dataset\n",
    "\n",
    "The dataset has to be preprocessed using the [preprocess_data_for_megatron.py](https://github.com/NVIDIA/NeMo/blob/main/scripts/nlp_language_modeling/preprocess_data_for_megatron.py) script included in the NeMo Framework. This step will also tokenize data using the `meta-llama/Meta-Llama-3.1-8B` tokenizer model to convert the data into a memory map format.\n",
    "\n",
    "> `NOTE:` In the block of code below, pass the paths to your train, test and validation data files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c49c1b8-2447-426c-9f24-bf5956aa2941",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!python /opt/NeMo/scripts/nlp_language_modeling/preprocess_data_for_megatron.py \\\n",
    "--input=\"./wikitext-data/wikitext-train.jsonl\" \\\n",
    "--tokenizer-library='huggingface' \\\n",
    "--tokenizer-type='meta-llama/Meta-Llama-3.1-8B' \\\n",
    "--output-prefix=wikitext_tokenized_train \\\n",
    "--append-eod \\\n",
    "--workers=32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72d14fd7-702f-4b74-a6e5-af3a60eef3a9",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!python /opt/NeMo/scripts/nlp_language_modeling/preprocess_data_for_megatron.py \\\n",
    "--input=\"./wikitext-data/wikitext-test.jsonl\" \\\n",
    "--tokenizer-library='huggingface' \\\n",
    "--tokenizer-type='meta-llama/Meta-Llama-3.1-8B' \\\n",
    "--output-prefix=wikitext_tokenized_test \\\n",
    "--append-eod \\\n",
    "--workers=32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1338a1ce-f0e2-4151-ad3d-d34db75ea1bd",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!python /opt/NeMo/scripts/nlp_language_modeling/preprocess_data_for_megatron.py \\\n",
    "--input=\"./wikitext-data/wikitext-val.jsonl\" \\\n",
    "--tokenizer-library='huggingface' \\\n",
    "--tokenizer-type='meta-llama/Meta-Llama-3.1-8B' \\\n",
    "--output-prefix=wikitext_tokenized_val \\\n",
    "--append-eod \\\n",
    "--workers=32"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb80e212-c343-4e51-a92d-184db43df011",
   "metadata": {},
   "source": [
    "After running the above scripts, you will see the preprocesed `wikitext_tokenized_{train/val/test}_text_document.{idx/bin}`files. These output files will be used in the next step."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9f30c0a-4315-4017-b014-add4291a3fde",
   "metadata": {},
   "source": [
    "\n",
    "### Step 2: Finetune the teacher on the dataset\n",
    "\n",
    "NeMo framework includes a standard python script [megatron_gpt_pretraining.py](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/megatron_gpt_pretraining.py) for training a model. Once you have your model downloaded and the dataset ready, fine-tuning the teacher model with NeMo is essentially just running this script!\n",
    "\n",
    "For this demonstration, this training run is capped by `STEPS`, and validation is carried out every `VAL_INTERVAL` steps.\n",
    "\n",
    "> `NOTE:` In the block of code below, pass the paths to your pre-processed train, test and validation data files as well as path to the teacher .nemo model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c31fd642-0304-43ed-9211-041dc36f22c3",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%bash \n",
    "\n",
    "export CUDA_DEVICE_MAX_CONNECTIONS=1\n",
    "\n",
    "\n",
    "# Set path(s) if different:\n",
    "\n",
    "MODEL=\"/workspace/llama-3_1-8b-instruct-nemo_v1.0/llama3_1_8b_instruct.nemo\"\n",
    "\n",
    "# Can change these to accommodate resources:\n",
    "\n",
    "TENSOR_PARALLEL_SIZE=8\n",
    "NODES=1\n",
    "MICRO_BATCH_SIZE=4\n",
    "\n",
    "# Don't change the following:\n",
    "\n",
    "EXPERIMENT_DIR=\"distill_trainings\"\n",
    "EXPERIMENT_NAME=\"megatron_llama_ft\"\n",
    "\n",
    "DATA_TRAIN='wikitext_tokenized_train_text_document'\n",
    "DATA_VAL='wikitext_tokenized_test_text_document'\n",
    "DATA_TEST='wikitext_tokenized_val_text_document'\n",
    "\n",
    "STEPS=30\n",
    "GLOBAL_BATCH_SIZE=128\n",
    "\n",
    "LOG_INTERVAL=1\n",
    "VAL_INTERVAL=10\n",
    "NUM_VAL_BATCHES=5\n",
    "\n",
    "LR=1e-4\n",
    "MIN_LR=1e-5\n",
    "WARMUP_STEPS=2\n",
    "\n",
    "\n",
    "cmd=\"torchrun --nproc-per-node=${TENSOR_PARALLEL_SIZE}\"\n",
    "\n",
    "${cmd} /opt/NeMo/examples/nlp/language_modeling/megatron_gpt_pretraining.py \\\n",
    "    --config-path /opt/NeMo/examples/nlp/language_modeling/conf/ \\\n",
    "    --config-name megatron_llama_distill.yaml \\\n",
    "    \\\n",
    "    name=${EXPERIMENT_NAME} \\\n",
    "    \\\n",
    "    exp_manager.exp_dir=${EXPERIMENT_DIR} \\\n",
    "    exp_manager.checkpoint_callback_params.save_top_k=1 \\\n",
    "    exp_manager.checkpoint_callback_params.save_nemo_on_train_end=True \\\n",
    "    \\\n",
    "    trainer.max_steps=${STEPS} \\\n",
    "    trainer.log_every_n_steps=${LOG_INTERVAL} \\\n",
    "    trainer.val_check_interval=${VAL_INTERVAL} \\\n",
    "    trainer.limit_val_batches=${NUM_VAL_BATCHES} \\\n",
    "    +trainer.num_sanity_val_steps=0 \\\n",
    "    \\\n",
    "    trainer.precision=bf16 \\\n",
    "    trainer.devices=${TENSOR_PARALLEL_SIZE} \\\n",
    "    trainer.num_nodes=${NODES} \\\n",
    "    \\\n",
    "    \"model.data.data_prefix={train:[1.0,$DATA_TRAIN],validation:[$DATA_VAL],test:[$DATA_TEST]}\" \\\n",
    "    \\\n",
    "    model.restore_from_path=${MODEL} \\\n",
    "    \\\n",
    "    ~model.tokenizer \\\n",
    "    +model.tokenizer='{library: huggingface, type: meta-llama/Meta-Llama-3.1-8B, use_fast: True}' \\\n",
    "    \\\n",
    "    model.tensor_model_parallel_size=${TENSOR_PARALLEL_SIZE} \\\n",
    "    model.sequence_parallel=True \\\n",
    "    model.micro_batch_size=${MICRO_BATCH_SIZE} \\\n",
    "    model.global_batch_size=${GLOBAL_BATCH_SIZE} \\\n",
    "    \\\n",
    "    model.encoder_seq_length=8192 \\\n",
    "    model.num_layers=32 \\\n",
    "    model.hidden_size=4096 \\\n",
    "    model.ffn_hidden_size=14336 \\\n",
    "    model.num_attention_heads=32 \\\n",
    "    model.hidden_dropout=0.0 \\\n",
    "    model.attention_dropout=0.0 \\\n",
    "    model.apply_query_key_layer_scaling=True \\\n",
    "    model.normalization='rmsnorm' \\\n",
    "    model.bias=False \\\n",
    "    model.activation='fast-swiglu' \\\n",
    "    model.position_embedding_type='rope' \\\n",
    "    model.share_embeddings_and_output_weights=False \\\n",
    "    model.num_query_groups=8 \\\n",
    "    ++model.scale_positional_embedding=True \\\n",
    "    ++model.rotary_base=500000.0 \\\n",
    "    \\\n",
    "    model.optim.name=distributed_fused_adam \\\n",
    "    model.optim.lr=${LR} \\\n",
    "    model.optim.sched.min_lr=${MIN_LR} \\\n",
    "    model.optim.sched.warmup_steps=${WARMUP_STEPS}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8aaf604a-efc0-4908-9055-5cf3bb0a05ae",
   "metadata": {},
   "source": [
    "This will create a finetuned teacher model named `megatron_llama_ft.nemo` in `./distill_trainings/megatron_llama_ft/checkpoints/`. We'll use this later.\n",
    "> `NOTE:`This script takes at least 20 minutes to run (depending on GPU) and will generate the finetuned teacher model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2709ccc0-bbb8-44ba-b00d-15b1dc5d60a7",
   "metadata": {},
   "source": [
    "### Step 3: Prune the finetuned-teacher model to create a student\n",
    "\n",
    "The next step is to trim the last 16 layers in the finetined teacher model. In this notebook, we are using depth-pruning and would be using the [megatron_gpt_drop_layers](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/megatron_gpt_drop_layers.py) script. \n",
    "> `NOTE:` In the block of code below, pass the paths to your finetuned teacher .nemo model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9715a1b-7a23-437f-b5e1-feec8e6c68e0",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!python -m torch.distributed.launch --nproc_per_node=8 \\\n",
    "     /opt/NeMo/examples/nlp/language_modeling/megatron_gpt_drop_layers.py \\\n",
    "     --path_to_nemo \"./distill_trainings/megatron_llama_ft/checkpoints/megatron_llama_ft.nemo\" \\\n",
    "     --path_to_save \"/workspace/4b_trimmed_model.nemo\" \\\n",
    "     --tensor_model_parallel_size 8 \\\n",
    "     --pipeline_model_parallel_size 1 \\\n",
    "     --gpus_per_node 8 \\\n",
    "     --drop_layers 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e9553db-9478-4074-9de1-1fa01a0e835c",
   "metadata": {},
   "source": [
    "Running this script will save the depth-pruned model `4b_trimmed_model.nemo` to your workspace."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8ada696-5d77-4113-9d15-a603113fdd58",
   "metadata": {},
   "source": [
    "\n",
    "### Step 4: Distill knowledge from teacher into student\n",
    "\n",
    "Distillation of a model with NeMo Framework is also possible using a python script: [megatron_gpt_distillation.py](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/megatron_gpt_distillation.py). \n",
    "\n",
    "For this demonstration, the `TEACHER` would be the finetuned teacher model `megatron_llama_ft.nemo` and the `STUDENT` model would be the pruned 4B model `4b_trimmed_model.nemo`. This training run is capped by `STEPS`, and validation is carried out every `VAL_INTERVAL` steps.\n",
    "\n",
    "> `NOTE:` In the block of code below, pass the paths to your pre-processed train, test and validation data files as well as path to the teacher and student .nemo models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61c0c69d-9401-4355-8725-78aa72eee8da",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%bash \n",
    "\n",
    "export CUDA_DEVICE_MAX_CONNECTIONS=1\n",
    "\n",
    "\n",
    "# Can change these to accommodate resources:\n",
    "\n",
    "TENSOR_PARALLEL_SIZE=8\n",
    "NODES=1\n",
    "MICRO_BATCH_SIZE=4\n",
    "\n",
    "# Don't change the following:\n",
    "\n",
    "EXPERIMENT_DIR=\"distill_trainings\"\n",
    "EXPERIMENT_NAME=\"megatron_llama_distill\"\n",
    "\n",
    "TEACHER=\"${EXPERIMENT_DIR}/megatron_llama_ft/checkpoints/megatron_llama_ft.nemo\"\n",
    "STUDENT=\"/workspace/4b_trimmed_model.nemo\"\n",
    "\n",
    "FINAL_MODEL_PATH=\"${EXPERIMENT_DIR}/${EXPERIMENT_NAME}/checkpoints/distilled_4b_model.nemo\"\n",
    "\n",
    "DATA_TRAIN='wikitext_tokenized_train_text_document'\n",
    "DATA_VAL='wikitext_tokenized_test_text_document'\n",
    "DATA_TEST='wikitext_tokenized_val_text_document'\n",
    "\n",
    "STEPS=30\n",
    "GLOBAL_BATCH_SIZE=128\n",
    "\n",
    "LOG_INTERVAL=1\n",
    "VAL_INTERVAL=10\n",
    "NUM_VAL_BATCHES=5\n",
    "\n",
    "LR=1e-4\n",
    "MIN_LR=1e-5\n",
    "WARMUP_STEPS=2\n",
    "\n",
    "\n",
    "cmd=\"torchrun --nproc-per-node=${TENSOR_PARALLEL_SIZE}\"\n",
    "\n",
    "${cmd} /opt/NeMo/examples/nlp/language_modeling/megatron_gpt_distillation.py \\\n",
    "    name=${EXPERIMENT_NAME} \\\n",
    "    \\\n",
    "    exp_manager.exp_dir=${EXPERIMENT_DIR} \\\n",
    "    exp_manager.checkpoint_callback_params.save_top_k=1 \\\n",
    "    \\\n",
    "    trainer.max_steps=${STEPS} \\\n",
    "    trainer.log_every_n_steps=${LOG_INTERVAL} \\\n",
    "    trainer.val_check_interval=${VAL_INTERVAL} \\\n",
    "    trainer.limit_val_batches=${NUM_VAL_BATCHES} \\\n",
    "    +trainer.num_sanity_val_steps=0 \\\n",
    "    \\\n",
    "    trainer.precision=bf16 \\\n",
    "    trainer.devices=${TENSOR_PARALLEL_SIZE} \\\n",
    "    trainer.num_nodes=${NODES} \\\n",
    "    \\\n",
    "    \"model.data.data_prefix={train:[1.0,$DATA_TRAIN],validation:[$DATA_VAL],test:[$DATA_TEST]}\" \\\n",
    "    \\\n",
    "    model.restore_from_path=${STUDENT} \\\n",
    "    model.kd_teacher_restore_from_path=${TEACHER} \\\n",
    "    model.nemo_path=${FINAL_MODEL_PATH} \\\n",
    "    \\\n",
    "    model.tensor_model_parallel_size=${TENSOR_PARALLEL_SIZE} \\\n",
    "    model.sequence_parallel=True \\\n",
    "    model.micro_batch_size=${MICRO_BATCH_SIZE} \\\n",
    "    model.global_batch_size=${GLOBAL_BATCH_SIZE} \\\n",
    "    \\\n",
    "    model.optim.name=distributed_fused_adam \\\n",
    "    model.optim.lr=${LR} \\\n",
    "    model.optim.sched.min_lr=${MIN_LR} \\\n",
    "    model.optim.sched.warmup_steps=${WARMUP_STEPS}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe7034ba-8c69-4edb-8c0f-84fdca43c152",
   "metadata": {},
   "source": [
    "This will create the final distilled model named `distilled_4b_model.nemo` in `./distill_trainings/megatron_llama_distill/checkpoints`.\n",
    "> `NOTE:`This script takes at least 35 minutes to run and generate the final distilled model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9a66d44-5028-47f9-9df3-9f07692e9461",
   "metadata": {},
   "source": [
    "### Step 5: Display the validation loss\n",
    "\n",
    "Now that the results are in, let's visualize the validation loss of the distilled model using the `tensorboard` library. \n",
    "> `NOTE:` This notebook demonstrates the use of the teacher finetuning, pruning and the distillation script. These scripts should ideally be run on a multi-node cluster with a larger `GLOBAL_BATCH_SIZE` and `STEPS` to see improvement in the validation loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be4da14c-c03f-4c28-accd-8f676dbef8a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext tensorboard\n",
    "%tensorboard --logdir \"distill_trainings/megatron_llama_distill/\" --port=6007"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08c63b80-0f24-4dde-b5d6-11db444726ed",
   "metadata": {},
   "source": [
    "Here is an image of the validation loss over 30 steps of running the training step in the distillation script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "648424fc-6a51-43ca-8f19-6ad05f949054",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://github.com/NVIDIA/NeMo/releases/download/r2.0.0rc1/val_loss_distillation.png\" width=\"400\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import Image, display\n",
    "display(Image(url=\"https://github.com/NVIDIA/NeMo/releases/download/r2.0.0rc1/val_loss_distillation.png\", width=400))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
